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(54) Environment recognition device provided with image pickup device 



(57) An environment recognition device formed in 
such a manner that an image sensor (2) is installed so 
that an image sensor mounting location recognition mark 
(5) falls within an image pickup area (4), means for com- 
paring mark locations the images of which are picked up 
sequentially by means of the image sensor (2) with 
respect to the initial location of the recognition mark (5) 
in the image pickup area (4) is provided, and, when it is 
detected that the mark location recognized sequentially 
has changed with respect to the initial location, an indi- 
cation signal for indicating that correction of the location 
of installing the image sensor (2) is required is outputted, 
thereby to prevent erroneous decision in image process- 
ing when the mounting position changes by vibration, 
contact or the like. 
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Description 

BACKGROUND OF THE INVENTION 

The present invention relates to an environment rec- s 
ognition device using an image pickup device, and to a 
device and a method for recognizing environment 
around a vehicle used in an on-vehicle device for recog- 
nizing external environment of a vehicle for supporting 
safety driving of a vehicle for instance and, to be con- io 
crete, using an image sensor such as a CCD camera as 
the image pickup device. 

Conventional environment recognition devices 
include, for example, one that is disclosed in JP-A-3- 
203000 that performs environment recognition using an 75 
image pickup device such as a CCD camera installed in 
a vehicle as an image sensor for the purpose of support- 
ing safety driving of the vehicle. Further, as disclosed in 
JP-A-1 -281600, a device for deciding whether there is a 
preceding vehicle within a safe distance or not based on 20 
image data of an image pickup device is also proposed. 
Since the picked up image data are different depending 
on the mounting location and the set angle of the image 
sensor in the environment recognition device using the 
image pickup device described above, uni vocal relative 25 
relationship between image data and recognition object 
information such as the distance between vehicles is 
adjusted after the image sensor is mounted and fixed. 
Or, the mounting location and the set angle of the image 
pickup device are adjusted so that the preset relative 30 
relationship between 'image data and the recognition 
object information may be obtained. 

In the conventional environment recognition device 
described above, however, compensation for the change 
with the passage of time of the mounting location and 35 
set angle of the image sensor has not been taken into 
consideration. As a result, there has been such a possi- 
bility that, even when the mounting location and the set 
angle of the image pickup device change with respect to 
the initialized location by vibration, contact or the like, an 40 
image processing means forms erroneous judgement 
based on inappropriate information of the image pickup 
device without noticing the change. 

A device in which a point is provided at a principal 
position on the road in a detection area is set based on as 
the position of this point in order to correct deviation of 
the detection area of a laser device as a vehicle detection 
device installed on the road has been disclosed in JP-A- 
5-62099. Further, a monitor for deciding the magnitude 
of the portion where brightness exceeds a critical value so 
by comparing image information inputted from an image 
sensor with reference image data, thereby to distinguish 
between the existence of an invader and deviation of the 
mounting location of the image pickup device by vibration 
or the like has been disclosed in JP-A-3-97080. In these ss 
prior arts, however, nothing has been described on the 
detection of deviation of the mounting location of an 
image sensor (an image pickup device), and further, the 
effects of the invention are influenced largely depending 



on what is the point of the former gazette and on what 
will be taken as the reference image data in the latter 
gazette. Therefore, appropriate setting of the point and 
the reference image data has been difficult. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to prevent erro- 
neous decision based on inappropriate information from 
an image pickup device when the mounting location of 
an image pickup device has changed with the passage 
of time due to vibration, contact or the like with respect 
to an initialized location. 

The present invention has been described with an 
environment recognition device of a vehicle as an exam- 
ple, but it can also be applied for preventing erroneous 
recognition due to deviation of a mounting location of an 
image pickup device when the environment is recog- 
nized from image pickup data in a general environment 
recognition device. 

In an environment recognition device of the present 
invention, an image pickup device is installed so that a 
recognition mark for specifying the initial mounting loca- 
tion of an image pickup device falls within an image 
pickup area in order to achieve the above-mentioned 
object. To be concrete, there is provided means for 
detecting and comparing location data of practical marks 
the images of which are picked up sequentially by the 
image pickup device with respect to the initial location 
data of the recognition mark in the image pickup area. 

Further, a mark showing a mounting location of an 
image pickup device is set up in a vehicle mounted with 
the image pickup device. The location data of the mark 
are inputted to a processing unit as one of peripheral 
information. 

In a method of recognizing environment from image 
data according to the present invention, the data showing 
the mounting location of an image pickup device are 
used as one of image processing data. 

In an environment recognition device of the present 
invention, a mark for specifying a mounting location of 
an image pickup device is set up in a portion that 
becomes within the image pickup area of the image sen- 
sor, and there are provided a memory portion for storing 
an initialized location of this mark and means for detect- 
ing and comparing the mark locations thereafter one 
after another in the image processing device. Further, 
when the sequentially recognized mark location has 
changed with respect to the initialized location, it is 
informed by an alarm or the like that the set location of 
the image sensor is inappropriate and requires correc- 
tion, or the posture of the image sensor is corrected auto- 
matically or the image data are corrected. 

Besides, in the present invention, the word "environ- 
ment" is used in a broad sense, and indicates all the 
objects that can be viewed as images. In the embodi- 
ment, it is a landscape around the vehicle, but it may be 
a work along a production line, cloths, or an indoor land- 
scape in other examples. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a schematic side view showing a rough 
structure of a first embodiment of the present inven- 
tion; 

Fig. 2 is a block diagram showing a structure of a 
signal processing portion in the first embodiment; 
Fig. 3 is a flowchart showing a processing operation 
of the embodiment; 

Fig. 4 is an image window showing the operation of 
the embodiment; 

Fig. 5 is a schematic side view showing a structure 
of another embodiment of the present invention; 
Fig. 6 is an image window showing the operation of 
the other embodiment; 

Fig. 7 is a schematic side view showing still another 
embodiment of the present invention; 
Fig. 8 is a block diagram showing a structure of a 
signal processing portion of the embodiment shown 
in Fig. 7; 

Fig. 9 is a flowchart showing signal processing oper- 
ation of the embodiment shown in Fig. 7; 
Fig. 1 0 is an image window for explaining the oper- 
ation of the above-mentioned embodiment; 
Fig. 11 is a schematic perspective view showing still 
another embodiment of the present invention; 
Fig. 1 2 is a block diagram showing a signal process- 
ing portion of still another embodiment of the present 
invention; 

Fig. 13 is a flowchart showing a signal processing 
operation of the embodiment shown in Fig. 12; 
Fig. 14 is a schematic perspective view showing still 
another embodiment of the present invention; and 
Figs. 1 5A and 1 5B show general views for explaining 
the operation of the embodiment shown in Fig. 14. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

Embodiments of the present invention will be 
described hereinafter with reference to the drawings. At 
the beginning, a general structure of an environment rec- 
ognition device for a vehicle that is an embodiment of the 
present invention will be shown with reference to Fig. 1. 
An image sensor 2 for picking up an image of environ- 
ment outside a vehicle is installed on its own vehicle 1. 
Further, a mark 5 is stuck on a windshield 3 so that the 
location thereof falls within an image pickup area 4 of the 
image sensor 2. Further, the image sensor 2 is con- 
nected to an image signal processing unit 7 by an image 
signal line 6, and the image signal processing unit 7 is 
also connected to a warning device 1 1, a brake control 
unit 12 and a steering control unit 13 through an alarm 
signal line 8, a brake control signal line 9 and a steering 
control signal line 10, respectively. 

Fig. 2 shows an internal structure of the image 
processing device 7, but it is divided roughly into five por- 
tions, i.e., an A/D converter 71 for converting an analog 
signal from the image sensor 2 into a digital signal, an 
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image memory 72 for taking in image data in a pixel unit, 
an image processing portion 73 for recognizing and com- 
puting a relative location of its own vehicle to the external 
environment, an image sensor location change deciding 

5 portion 74 for detecting and deciding location deviation 
of the image sensor with respect to the initialized loca- 
tion, and a danger avoidance deciding portion 75 for 
deciding control of issuance of warning, brake and steer- 
ing. The image processing portion 73 is composed of a 

w preceding vehicle recognition portion 731, a distance 
between vehicles arithmetic portion 732, an own vehicle 
lane recognition portion 733 and a location relationship 
between own vehicle and lane arithmetic portion 734. 
The image sensor location change deciding portion 74 

75 is composed of an initial mark location storage portion 
741 connected to an initial mark location setting device 
1 4, a mark extracting portion 742, a mark location detect- 
ing portion 743 and a comparator 744. The danger avoid- 
ance deciding portion 75 is composed of a risk deciding 

20 portion 751, a warning issuance deciding unit 752 and 
an actuator drive deciding unit 753. 

Next, the operation of the present embodiment will 
be described with reference to Fig. 2 to Fig. 4. In Fig. 2, 
the image data in front of its own vehicle picked up by 

25 the image sensor 2 are converted into 8-bit digital data 
for instance by means of the A/D converter 71 and stored 
in the image memory 72. Recognition of a preceding 
vehicle (731) and computation of the distance from the 
preceding vehicle (732) are performed in the image 

30 processing portion based on these data. Further, recog- 
nition of its own vehicle lane (733) and computation of 
location relationship between its own vehicle and the 
lane (734) are performed in parallel therewith. Abnormal 
approach to a preceding vehicle, lane jutting out running 

35 and so on are decided in the risk deciding portion 751 
based on these results of computation, and when it is 
decided that collision can be avoided by informing a 
driver of the fact with warning, a warning for attracting a 
driver's attention to the collision avoidance operation ts 

40 given from the warning device 1 1 by the decision of the 
warning issuance deciding unit 752. On the other hand, 
when it is decided in the risk deciding portion 751 that 
collision is unavoidable by warning only, brake control 
(1 2) and/or steering control (1 3) is performed by the deci - 

45 sion of the actuator drive deciding unit 753, thereby to 
perform collision avoidance on the car side without oper- 
ation by the driver. 

The foregoing shows general operation of the colli- 
sion avoidance unit, but a characteristic portion of the 

so present embodiment will be described with reference to 
Fig. 2 to Fig. 4 in the next place. When initialization of 
the image sensor is completed, the initial mark location 
setting device 14 is operated (1001) so as to store the 
location of the mark 5 in the initial mark location storage 

55 portion 741 in the image sensor location change decid- 
ing portion 74 (1002). At this time, a reloadable ROM or 
a RAM having a backup power supply is used in the initial 
mark location storage portion 741 so that the memory of 
the initial mark location does not disappear even when 
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turning on electricity to the image signal processing unit 
is suspended because of the reason that a battery cable 
is disconnected and so on. 

When setting of the initial mark location is com- 
pleted, the following operation is performed thereafter in 
parallel with the normal collision avoidance operation 
described above. Namely, the location of the mark 5 dur- 
ing running is detected from the image data taken into 
the image memory 72 through the mark extracting por- 
tion 742 and the mark location detecting portion 743 
(1 003). Next in the comparator 744, the location is com- 
pared with the data stored in the initial mark location stor- 
age portion 741 (1004). Here, when the mark location 
detected sequentially is equal to the initial mark location 
(1 005), it is decided that the location of the image sensor 
has not been changed, processings described above in 
the image processing portion 73 and the danger avoid- 
ance deciding portion 75 are performed (1006 and 
1007), and the process is returned to the operation of 
taking in the image of the next frame. 

On the other hand, when two mark locations are dif- 
ferent from each other as shown at 5 (an initial position) 
and 5' (a detected location during running) in the image 
window 15 shown in Fig. 4 in the processing (1005) in 
the comparator 744, it is decided that the image sensor 
has changed with respect to the initialized location 
(1008), and a warning is given to the driver through the 
warning issuance deciding unit 752 and the warning 
device 1 1 (1 009). At this time, the warning is given to the 
driver while changing the tone color or the like so as not 
to confuse with the collision avoidance warning such as 
preceding vehicle abnormal approach and car lane jut- 
ting out. Further, transmission of an image signal from 
the image memory 72 to the image processing portion 
73 is stopped, and the danger avoidance deciding func- 
tion is suspended until the image sensor location is read- 
justed to the initial location. 

According to the present embodiment, it is possible 
to always monitor the location change of the image sen- 
sor by extracting the mark location out of image data 
taken in sequentially and comparing this mark location 
with the initial location. As a result, it is possible to pre- 
vent such a problem that an erroneous distance between 
vehicles is computed using a distance A* in an image win- 
dow in a state that the image sensor is deviated, thus 
giving inappropriate risk decision when risk decision is 
ought to be made originally by computing the distance 
from a preceding vehicle using a distance A in the image 
window as shown in Fig. 4. 

Next, a second embodiment of the present invention 
will be described with reference to Fig. 5 and Fig. 6. In 
the present embodiment, a bonnet mascot 16 fitted to 
the bonnet of its own vehicle is utilized in place of the 
mark 5 stuck to the windshield 3 in the first embodiment. 
In the present embodiment, the image sensor 2 is 
installed so that the bonnet mascot 16 is located within 
an image pickup area 4 thereof, and the initialized loca- 
tion and a detected location 16' during running of the 
bonnet mascot 16 within the image window 15 such as 
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shown in Fig. 6 are compared sequentially with each 
other, thereby to monitor and decide the location change 
of the image sensor. In the structure and the operation, 
the mark 5 is only replaced with the bonnet mascot 16 

5 with respect to the first embodiment shown in Fig. 1 to 
Fig. 3, and entirely similar effects asthefirst embodiment 
are obtainable. Furthermore, since the operation for 
sticking the mark 5 later becomes unnecessary in the 
present embodiment, it is possible to reduce mandays 

w as compared with the first embodiment. 

The bonnet mascot fitted to the bonnet of its own 
vehicle has been utilized in the present embodiment, but 
it is a matter of course that similar effects are obtainable 
when any of a part of the bonnet, an automobile inspec- 

15 tion seal pasted to the windshield, a part of ceiling or the 
like that is fixed to its own vehicle is used in place of the 
bonnet mascot. 

Next, a third embodiment of the present invention 
will be described with reference to Fig. 7 to Fig. 10. What 

20 differs from the first embodiment in the structure of the 
present embodiment is a fact that the image sensor 2 is 
fitted to its own vehicle 1 through a universal joint 1 7 that 
is able to be turned freely in a vertical direction R b and 
in a horizontal direction R a by means of a motor as shown 

25 in Fig. 7, and furthermore, is a fact that the comparator 
744 and the joint 17 are connected through an image 
sensor direction control unit 76 as shown in Fig. 8. 

The operation of the present embodiment becomes 
similar to that of the first embodiment shown in Fig. 3 

30 when deviation of a mark from the initial location is not 
detected in the comparator 744 as shown in the flow 
1 002 to 1 007 in Fig. 9. On the other hand, when the devi- 
ation is detected (1 005), it is decided in the image sensor 
direction control unit 76 whether the deviation quantity is 

35 correctable or not (1010). When correction is possible, it 
is computed in which direction and to what extent the 
angle of the image sensor is to be changed based on the 
magnitudes a and b in the horizontal direction and the 
vertical direction of the mark deviation quantity shown in 

40 Fig. 10 (101 1 ), and the universal joint 1 7 is driven by the 
motor, thereby to correct so that the mark location 5' coin- 
cides with the initial location 5 (1012). However, when it 
is decided in the processing 1010 that the required cor- 
rection quantity in the image sensor direction is out of 

45 the movable range of the joint 17 (1013), a warning is 
given to the driver through the warning issuance deciding 
unit 752 and the warning device 11 (1014). 

In the present embodiment, when deviation of the 
image sensor is produced, it is corrected automatically 

so on the side of the device. Therefore, an effect that the 
driver is released from readjustment work is obtainable 
in addition to the effect of the first embodiment. 

Next, a fourth embodiment of the present invention 
will be described with reference to Fig. 1 1 . The present 

55 embodiment is similar to the third embodiment in that the 
direction of the image sensor is corrected automatically, 
but differs therefrom in a point that only an image pickup 
portion 210 consisting of a lens 201 and a CCD element 
202 is driven by means of a motor contained in the image 
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sensor 2 without changing the direction of the whole 
image sensor by means of the joint 1 7. 

In the present embodiment, since the weight of the 
driving portion is reduced substantially, the response 
speed of the image pickup system to a correction signal 5 
is improved and the safety is improved further in addition 
to the effect of the third embodiment. 

Next, a fifth embodiment of the present invention will 
be described with reference to Fig. 10, Fig. 12 and Fig. 
1 3. What differs from the first embodiment in point of the 
structure of the present embodiment is the fact that an 
image data correcting portion 77 for performing correc- 
tion and storage of data is provided between the image 
memory 72 and the image processing portion 73 as 
shown in Fig. 12, thereby to feed back the result of the 
comparator 744 to the image data correcting portion 77. 

The operation of the present embodiment becomes 
similar to that in the first embodiment shown in Fig. 3 
when deviation of the mark with respect to the initial loca- 
tion is not detected in the comparator 744 as shown in 
the flow from 1 002 to 1 007 in Fig. 1 3. On the other hand, 
when deviation is detected (1005), it is decided in the 
image correcting portion 77 whether the deviation quan- 
tity is correctable in the image window or not (1015). 
When correction is possible, it is computed in which 
direction and to what extent the image data are to be 
changed based on the magnitudes a and b in a horizontal 
direction and a vertical direction of mark deviation quan- 
tities a and b shown in Fig. 10 (1016), and the image data 
are corrected based on the computed quantities (101 7). 
For example, as a simple correction method, there is a 
method of adding to or subtracting from all the data in 
the field of view vertically or horizontally by the pixel por- 
tion that the mark is deviated. Thereafter, the process- 
ings thereafter (1006, 1007) are performed using 
corrected data. When it is decided in the process 1015 
that the required correction quantity of the image data is 
out of the movable range in the image window (1018), 
however, a warning is given to a driver through the warn- 
ing issuance deciding unit 752 and the warning device 
11 (1019). 

In the present embodiment, similar effects as those 
in the third and the fourth embodiments are obtainable, 
but a mechanism for rotating the image sensor becomes 
no longer required, thus producing an effect that weight 
and cost are reduced and reliability is also improved 
remarkably in addition to the above. 

Next, a sixth embodiment will be described with ref- 
erence to Fig. 14 and Figs. 15A and 15B. In Fig. 14, a 
state V in which a vehicle jams on the brakes is drawn 
one over the other by comparing with a normal running 
state 1 , but an image sensor 2' installed in vehicle is 
changed with respect to a normal position 2 in this state. 
Further, a posture change of this kind is also produced 
in running a bad road such as shown in Fig. 15B. As a 
result, the image pickup area also changes from 4 to 4', 
and the relative relationship between a location of a run- 
ning vehicle and a practical distance between vehicles 
in the image window changes, thus causing erroneous 
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judgement on the risk. In the present embodiment, the 
posture change of a vehicle is detected based on the 
information of height sensors 18a and 18b, an acceler- 
ometer 19 or the like arranged in front and in the rear of 
the vehicle. When a posture change is detected, it is 
decided that there is a possibility of erroneous judgement 
on the risk, and the transmission of an image signal from 
the image memory 72 to the image processing portion 
73 in the image signal processing unit 7 shown in Fig. 2 
is suspended. Then, when such a state is calmed down 
and the posture change is also calmed down, the image 
signal transmission from the image memory 72 to the 
image processing portion 73 is restarted. 

In the present embodiment, being different from 
those embodiments that have been described so far, it 
is possible to prevent occurrence of erroneous decision 
on the risk that could occur even when the direction of 
the image sensor does not change with respect to initial- 
ization. 

According to the present invention, an image pickup 
device has been installed so that a mark for specifying a 
mounting location of the image pickup device falls within 
the image pickup area as described in detail above. 
Thus, it is possible, even when a recognition location of 
the image pickup device changes as time elapses with 
respect to the initialized location, to prevent erroneous 
decision of the image processing unit based on the 
change. To be concrete, in an environment recognition 
device for a vehicle, it is possible to give a proper decision 
on a distance between vehicles for instance by providing 
means for recognizing and comparing mark locations 
picked up sequentially by an image pickup device with 
respect to the initial location of the mark in an image 
pickup area, and by informing by an alarm or the like a 
driver that the installation location of the image pickup 
device is inappropriate and required to be corrected, cor- 
recting the mounting location of the image pickup device 
automatically or correcting image data when the mark 
locations recognized sequentially change with respect to 
the initialized location. 

Claims 

1 . An environment recognition device using an image 
pickup device for recognizing environment by signal 
processing of image pickup data obtained from an 
image pickup device (2), characterized in that there 
is provided a mark (5) for specifying a location of 
mounting said image pickup device (2) in an image 
pickup area (4) of said image pickup device (2). 

2. A vehicle provided with an image pickup device (2) 
for obtaining peripheral environmental information 
as image data, characterized in that there is pro- 
vided a mark (5) for giving mounting location data of 
said image pickup device (2) as one of said image 
data in an image pickup area (4). 
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3. An image pickup device for a vehicle mounted on a 
vehicle (1) and outputting environmental information 
inside and outside the vehicle (1) as image data 
characterized in that a mark (5) showing a location 7. 
tor mounting said image pickup device (2) is installed 5 

so as to fall within an image pickup area (4) thereof. 

4. A method of recognizing environment from image 
pickup data by image processing of image pickup 
data picked up by an image pickup device (2), char- 10 
acterized in that the data showing the location of 
mounting said image pickup device (2) are used for 
detecting deviation of a field of view of the image 
pickup device (2). 

15 

5. An environment recognition device for a vehicle 
using an image pickup device having an image 
pickup device (2) installed in vehicle for picking up 
an image of the exterior of a vehicle (1), image 
processing means (7) for deciding existence of a 20 
state bringing a danger to said vehicle (1) by using 
image pickup data of said image pickup device (2), 
and warning means (11), when a state that brings a 
danger is recognized by said image processing 
means (7) , for al erting a driver to that effect, charac- 25 
terized in that a mark (5) is stuck to a fixed portion 

of a vehicle (1 ) that falls within an image pickup area 

(4) of said image pickup device (2), memory means 8. 

(72) for storing an initial location of said mark (5) and 

detecting means (74) for detecting said mark I oca- 30 

tion sequentially while the device is in operation are 

provided in said image processing means (7), and, 

when the mark location detected sequentially by 

said detecting means (74) has changed with respect 

to the initial location of said mark (5), a driver is 35 

alerted to that effect through said warning means 

(11). 

6. An environment recognition device for a vehicle 
using an image pickup device having an image 40 
pickup device (2) installed in vehicle for picking up 

an image of the exterior of a vehicle (1), image 
processing means (7) for deciding existence of a 
state bringing a danger to said vehicle (1) by using 
image pickup data of said image pickup device (2), 45 9. 
and warning means (11), when a state that brings a 
danger is recognized by said image processing 
means (7), for alerting a driver to that effect, charac- 
terized in that a mark (5) is stuck to a fixed portion 
of a vehicle ( 1 ) that falls within an image pickup area so 
(4) of said image pickup device (2) t memory means 
(72) for storing the initial location of said mark (5) 
and detecting means (74) for detecting said mark 
location sequentially while the device is in operation 
are provided in said image processing means (7), 55 
and an image signal is continued to be transmitted 
from said image pickup device (2) to said image 
processing means (7) except for the change of the 
mark location detected sequentially by said detect- 



ing means (74) with respect to the initial location of 
said mark (5). 

An environment recognition device for a vehicle 
using an image pickup device having an image 
pickup device (2) installed in vehicle for picking up 
an image of the exterior of a vehicle (1). image 
processing means (7) for deciding existence of a 
state bringing a danger to said vehicle (1) by using 
image pickup data of said image pickup device (2), 
and warning means (11). when a state that brings a 
danger is recognized by said image processing 
means (7), for alerting a driver to that effect, charac- 
terized in that a mark (5) is stuck to a fixed portion 
of a vehicle (1 ) that falls within an image pickup area 
(4) of said image pickup device (2), memory means 
(72) for storing the initial location of said mark (5) 
and detecting means (74) for detecting said mark 
location sequentially while the device is in operation 
are provided in said image processing means (7), 
and an image signal transmission from said image 
pickup device (2) to said image processing means 
(7) is suspended when the mark location detected 
sequentially by said detecting means (74) has 
changed with respect to the initial location of said 
mark (5). 

An environment recognition device for a vehicle 
using an image pickup device having posture 
change sensors (18a, 18b) for sensing a posture 
change of a vehicle, an image pickup device (2) 
installed in vehicle for picking up the image of the 
exterior of a vehicle, image processing means (7) for 
deciding the existence of a state that brings a danger 
to said vehicle (1) using image data of said image 
pickup device (2), and alerting means, when a state 
that brings a danger is recognized by said image 
processing means (7), for alerting a driver to that 
effect, characterized in that, when a posture change 
of a vehicle (1 ) is sensed by said posture change 
sensors (18a, 18b), the image signal transmission 
from said image pickup device (2) to said image 
processing means (73) is suspended. 

An environment recognition device for a vehicle 
using an image pickup device having an image 
pickup device (2) installed in vehicle for picking up 
an image of the exterior of a vehicle (1), image 
processing means (7) for deciding existence of a 
state bringing a danger to said vehicle (1) by using 
image pickup data of said image pickup device (2), 
and warning means (11), when a state that brings a 
danger is recognized by said image processing 
means (7). for alerting a driver to that effect, charac- 
terized in that said image pickup device (2) is 
installed within the image pickup area (4) of said 
image pickup device (2) so that the fixed portion of 
a vehicle (1 ) is picked up, memory means (72) for 
storing the initial location of the fixed portion of said 
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vehicle (1) and detecting means (74) for detecting 
fixed portion of said vehicle sequentially during the 
operation of the device are provided in the image 
processing means (7), and, when the location of the 
fixed portion of the vehicle detected sequentially by 5 
detecting means (74) has changed with respect to 
the initial location of the fixed portion of said vehicle 
(1), a driver is alerted to that effect through said 
warning means (11). 

10 

10. An environment recognition device for a vehicle 
using an image pickup device according to Claim 9. 
wherein the fixed portion of a vehicle (1) is a hood 
in front of a windshield (3). 

75 

11. An environment recognition device for a vehicle 
using an image pickup device according to Claim 9, * 
wherein the fixed portion of a vehicle (1) is a seal 
stuck on a windshield (3). 

20 

12. An environment recognition device for a vehicle 
using an image pickup device according to Claim 9, 
wherein the fixed portion of a vehicle (1) is a trunk 
hood in the rear of a rear windshield. 

25 

13. An environment recognition device for a vehicle 
using an image pickup device having an image 
pickup device (2) installed in vehicle for picking up 
an image of the exterior of a vehicle (1), image 
processing means (7) for deciding existence of a 30 
state bringing a danger to said vehicle (1) by using 
image pickup data of said image pickup device (2), 
and warning means (11), when a state that brings a 
danger is recognized by said image processing 
means (7), for alerting a driver to that effect, charac- 35 
terized in that said image pickup device (2) is fixed 

in vehicle through a direction control unit (76) that is 
able to change an image pickup direction thereof 
and a mark (5) is stuck on a fixed portion of a vehicle 
(1) that falls within an image pickup area (4) of said 40 
image pickup device (2), memory means (72) for 
storing the initial location of said mark (5) and detect- 
ing* means (74) for detecting said mark location 
sequentially while the device is in operation are pro- 
vided in said image processing means (7), and, 45 
when the mark location detected sequentially by 
said detecting means (74) has changed with respect 
to the initial location of said mark (5), the direction 
of said image pickup device (2) is adjusted automat- 
ically so that the mark location coincides with the ini- 50 
tial location of said mark (5) by driving the direction 
control unit (76) of said image pickup device (2). 

14. An environment recognition device for a vehicle 
using an image pickup device having an image 55 
pickup device (2) installed in vehicle for picking up 

an image of the exterior of a vehicle (1), image 
processing means (7) for deciding existence of a 
state bringing a danger to said vehicle (1) by using 
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image pickup data of said image pickup device (2). 
and warning means (11), when a state that brings a 
danger is recognized by said image processing 
means (7), for alerting a driver to that effect, charac- 
terized in that a mark (5) is stuck to a fixed portion 
of a vehicle (1) that falls within an image pickup area 
(4) of said image pickup device (2), memory means 
(72) for storing the initial location of said mark (5) 
and detecting means (74) for detecting said mark 
location sequentially while the device is in operation 
are provided in said image processing means (7), 
and. when the mark location detected sequentially 
by said detecting means (74) has changed with 
respect to the initial location of said mark (5), the 
image data taken into said image processing means 
(7) are corrected in accordance with the change 
quantity and the direction of the change. 

15. An environment recognition device for a vehicle 
using an image pickup device according to Claim 5, 
wherein memory means (72) for storing an initial 
location of a mark (5) is a read-only memory (ROM) 
that is rewritable from the outside. 

16. An environment recognition device for a vehicle 
using an image pickup device according to Claim 5, 
wherein memory means (72) for storing an initial 
location of a mark (5) is a random access memory 
(RAM) having a power source for backing up so that 
the memory is not erased even when an engine is 
stopped. 

17. An environment recognition device for a vehicle 
using an image pickup device according to Claim 13, 
wherein direction control of the image pickup device 
(2) is made by a motor. 

18. An environment recognition device for a vehicle 
using an image pickup device according to Claim 13, 
wherein it is only the image pickup system (210) in 
an image pickup device composed of a lens (201) 
and a charge coupled device (CCD) element (202) 
that the direction changes. 

19. An environment recognition device for a vehicle 
using an image pickup device according to Claim 8, 
wherein a posture change of a vehicle (1) is pre- 
sumed by outputs of a height sensors (1 8a, 1 8b) pro- 
vided on a suspension. 

20. An environment recognition device for a vehicle 
using an image pickup device according to Claim 8, 
wherein a posture change of a vehicle (1) is pre- 
sumed by an output of an accelerometer (19) fitted 
on a vehicle body. 



7 



BNSOOCID: <EP 06915&9A2_I_> 



EP 0 691 599 A2 





BNSDOCID: <EP 0691 599A2_I_> 



EP 0 691 599 A2 



F I G. 3 



START 



SET INITIAL MARK LOCATION 



— IOOI 



STORE INITIAL 
MARK LOCATION 



DETECT MARK LOCATION 



-I002 



— 1003 



COMPARE WITH 
INITIAL LOCATION 




IS MARK LOCATION 
EQUAL TO INITIAL 
LOCATION ? 



YES ^ 



IMAGE PROCESSING 



I 



-1004 




1005 



l 



L 



1008 



DECIDE THAT IMAGE SENSOR 
HAS CHANGED WITH RESPECT TO 
INITIALISED LOCATION 



^1006 



DANGER AVOIDANCE DECISION 



WARNING 



1007 



r 



1009 



10 

BNSDOCID: <EP 0691 5S9A2_I_> 



BNSDOCtD: <EP 0691S99A2J_> 



EP 0 691 599 A2 



FIG. 5 




BNSDOCID: <EP 0691 599A2J_> 



12 



EP 0 691 599 A2 



F I G. 6 




EP0 691 599 A2 



F I G . 7 




14 

BNSDOCID: <EP 0691599A2_I_> 



EP 0 691 599 A2 



FIG. 9 



STORE INITIAL 
MARK LOCATION 



I002 



DETECT MARK LOCATION 



^1003 



COMPARE WITH 
INITIAL LOCATION 



I 



IS MARK LOCATION\NO 
EQUAL TO INITIAL 
LOCATION ? 



YES 




1005 



IMAGE PROCESSING 



1006 



DANGER AVOIDANCE 
DECISION 



TTi007 



DECIDE CORRECTION 
IMPOSSIBLE 



1013 



WARNING 




^^014 



DEVIATION \ MO 
CORRECTABLE ^ 
AREA? 



I 



YES 




1010 



COMPUTE CORRECTION 
ANGLES( Ra . Rb ) 



I 



101 



CORRECT DIRECTION OF 
IMAGE SENSOR 



1012 



16 

BNSDOCID: <EP 0691599A2_I_> 



EP 0 691 599 A2 



F I G. 10 




17 

BNSDCCID: <EP 0691599A2J_> 



EP 0 691 599 A2 



F I G . 1 1 




18 

BNSDOCID: <EP 0691 599A2J _> 



EP O 691 599 A2 



CD 



LU 
O 

^ > 



S- 1 



GO 



SI 



I 

I 

i 

! 
I 
I 



3 



C\J 



ro 



8 



coo 















WARM 

ISSUA 
DECO 
UMT 




TUATC 
CIDIN6 









<->? 



I 



I - 



~T1 
_l i< 



IS 



ro 
m 



._J N- 



^ 8, 



iro 

IN- 







* §. 



rO 



J 



3 1 



I 1 4- — 



OJ 



if 



Q 
< 



*4-J 



ro 



.N- 

r- 



-CM 




EP 0 691 599 A2 



FIG. 13 



STORE INITIAL 
MARK LOCATION 



I 



DETECT MARK LOCATION 

I ^H003 



COMPARE WITH 
INITIAL LOCATION 



jf M004 

/Is MARK LOCATION \NO 
< EQUAL TO INITIAL 
\LOCATION ? 



YES 



I 



1005 



IMAGE PROCESSING 



1006 



DANGER AVOIDANCE 
DECISION 



^07 









DECIDE CORRECTION 
IMPOSSIBLE 




I M0I8 


WARNING 






V I0I9 


, 


DEVIATION 
< CORRECTABLE N 


NO 



AREA? 



I 



YES 



1015 



COMPUTE IMAGE DATA 
CORRECTION QTY. 



1016 



IMAGE CORRECTION 



1017 



20 

B.MSOOCIO; <EP 0691S99A2J_> 



EP 0 691 599 A2 

FIG. !4 




FIG. I5B 



21 



BNSDOCID: <EP 0691599A2J_> 



^ge Blank (uspto) 



(19) 



J 



(12) 



Europaisches Patentamt 
European Patent Office 
Off ice europeen des brevets (11) EP 0 691 599 A3 

EUROPEAN PATENT APPLICATION 



(88) Date of publication A3. 

06.03.1 996 Bulletin 1 996/1 0 

(**o) Date ot puDii canon h\z. . 

1 u.ui .i sjsjo ouiietin i yyo/u.*: 

^ i ) Application nurnuer. ^oouoroo.s 

(22> Date of filina 02.06.1995 


(51) Int. CI. : vaUoD l/Uo 


(84) Designated Contracting States: 


• Tanaka, Yasunari 


DE FR GB 


Mito-shi (JP) 




* Ondoh, Eiryoh 


(30) Priority: 05.07.1994 JP 153288/94 


Hitachi naka-shi (JP) 




• Shioya, Ma koto 


(71) Applicant: HITACHI, LTD. 


Tokyo (JP) 


Chiyoda-ku, Tokyo 100 (JP) 






(74) Representative: Molyneaux, Martyn William 


(72) Inventors: 


Langner Parry 


* Takano, Kazuaki 


52-54 High Holborn 


Mito-shi (JP) 


London WC1V6RR (GB) 


• Monzi, Tatsuhiko 




Hitachinaka-shi (JP) 





(54) Environment recognition device provided with image pickup device 



(57) An environment recognition device formed in 
such a manner that an image sensor (2) is installed so 
that an image sensor mounting location recognition mark 
(5) falls within an image pickup area (4), means for com- 
paring mark locations the images of which are picked up 
sequentially by means of the image sensor (2) with 
respect to the initial location of the recognition mark (5) 
in the image pickup area (4) is provided, and, when it is 
detected that the mark location recognized sequentially 
has changed with respect to the initial location, an indi- 
cation signal for indicating that correction of the location 
of installing the image sensor (2) is required is outputted, 
thereby to prevent erroneous decision in image process- 
ing when the mounting position changes by vibration, 
contact or the like. 



FIG. I 



CO 
< 

LO 

<T> 
CD 

o 
a. 

LU 



BNSOOCID: <EP 069 1599 A3 J _> 




Printed by Rank Xerox (UK) Business Services 
2.9.12/3.4 



EP 0 691 599 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 95 30 3783 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document wicb indication, where appropriate, 
of relevant passages 



Relevant 

to 



CLASSIFICATION OF THE 
APPLICATION (lnt-CJ-6) 



EP-A-0 132 151 (TOYOTA) 

* page 5, line 6 - page 8, line 11 * 

PROCEEDINGS OF THE 1992 IEEE/RSJ 

INTERNATIONAL CONFERENCE ON INTELLIGENT 

ROBOTS AND SYSTEMS, 

vol .3, 7 July 1992, USA 

pages 2195 - 2202 

H.ISHIGURO ET AL 'ACTIVE VISION BY 

MULTIPLE VISUAL AGENTS' 

* the whole document * 

US-A-3 882 268 (Y.OGAWA ET AL) 

* the whole document * 



1-4 



1-9,13, 
14 



G05D1/03 



1-9,13, 
14 



TECHNICAL FIELDS 
SEARCHED (lnuCL6) 



G05D 



The present search report has been drawn up for all dairas 

Ptan •< u«tk Date of csmdMIdi 



THE HAGUE 



Date of OMpWtei «f tat uvck 

4 December 1995 



Kel peris, K 



3 
| 



CATEGORY OF CITED DOCUMENTS 

X : particularly relevant if taken alone 

Y : particularly relevant if comibtned with another 

document of the same category 
A : technological background 
O : non-written disclosure 
P : intermediate document 



T : theory or principle underlying the invention 
E : earlier patent document, but published on, or 

after the riling dale 
D : document cited io the application 
L : docusent cited for other reasons 



A : member of the same patent family, corresponding 
document 



BNSOOCID: <EP 0691599Aj J_> 



EP 0 691 599 A3 




European Patent 
Office 



CLAIMS INCURRING FEES 



The present European patent appfication composed at the lime of fifing more than en daims. 

□ 



AH claims toes have been paid withm toe prescribed time limit The present European search report has been 
drawn up for ait dams. 



PI Onty part of the dams tees have been paid within the prescribed time ImiL The present European search 
LI ropon nas ^en Mwn up far me first ten dams and for those dams tor which dams tees haw been paid. 



□ 



namely dams: 

No dams tees have been paid within the prescribed time limit. The present European search report has been 
drawn up tor the first ten darns. 



LACK OF UNITY OF INVENTION 

The Search Division considers thai the present European patent application does not comply with the retirement of unity 

nvenfton and relates to several inventions or groups of inventions. 

namely: 



0 



see sheet -B- 



Afl further search lees have been pad within the fixed time ImrL The present European search report has 
been drawn up tor all dams. 



r— i Onry part of the further search fees have been paid within toe fund time limit. The present European search 
l—l report has been drawn up to r those parts of the European patent application which relate to the inventions in 



respects of which search tees have been paid, 
namely dams: 



r— i None of the further search tees has been paid within the fixed time ImrL The present European search report 
' — ' has been drawn up for those parts of the European patent application which relate to the invention first 



mentioned n the daims. 
namety dams: 



3 

BNSDOCID: <EP 0691 599A3_I_> 



EP 0 691 599 A3 



3 



Office 



European Patent 



EP 95 30 3783 -B- 



LACK OF UNITY OF INVENTION 



The Search Division considers that the present European patent application does not comply with the requirement of unity of 
invention and relates to several inventions or groups of inventions, 
namety : 



1. Claims 1-4,13,17,18: 

Environment recognition device characterised by a 
mark for specifying location of mounting 

2. Claims 5-12,14-16,19,20: 

Environment recognition device having a memory 
and warning means 



BNSDCCID: <EP 0691 599A3_I_> 



4 



